In this notebook, we're going to use all the techniques we've learned thus far to perform neural network training (and prediction) while both the model and the data are encrypted.
In particular, we present our custom Autograd engine which works on encrypted computations.
Authors:
In [ ]:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import syft as sy
In [ ]:
# Set everything up
hook = sy.TorchHook(torch)
alice = sy.VirtualWorker(id="alice", hook=hook)
bob = sy.VirtualWorker(id="bob", hook=hook)
james = sy.VirtualWorker(id="james", hook=hook)
In [ ]:
# A Toy Dataset
data = torch.tensor([[0,0],[0,1],[1,0],[1,1.]])
target = torch.tensor([[0],[0],[1],[1.]])
# A Toy Model
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.fc1 = nn.Linear(2, 2)
self.fc2 = nn.Linear(2, 1)
def forward(self, x):
x = self.fc1(x)
x = F.relu(x)
x = self.fc2(x)
return x
model = Net()
Encryption here comes in two steps. Since Secure Multi-Party Computation only works on integers, in order to operate over numbers with decimal points (such as weights and activations), we need to encode all of our numbers using Fixed Precision, which will give us several bits of decimal precision. We do this by calling .fix_precision().
We can then call .share() as we have for other demos, which will encrypt all of the values by sharing them between Alice and Bob. Note that we also set requires_grad to True, which also adds a special autograd method for encrypted data. Indeed, since Secure Multi-Party Computation doesn't work on float values, we can't use the usual PyTorch autograd. Therefore, we need to add a special AutogradTensor node that computes the gradient graph for backpropagation. You can print any of this element to see that it includes an AutogradTensor.
In [ ]:
# We encode everything
data = data.fix_precision().share(bob, alice, crypto_provider=james, requires_grad=True)
target = target.fix_precision().share(bob, alice, crypto_provider=james, requires_grad=True)
model = model.fix_precision().share(bob, alice, crypto_provider=james, requires_grad=True)
In [ ]:
print(data)
In [ ]:
opt = optim.SGD(params=model.parameters(),lr=0.1).fix_precision()
for iter in range(20):
# 1) erase previous gradients (if they exist)
opt.zero_grad()
# 2) make a prediction
pred = model(data)
# 3) calculate how much we missed
loss = ((pred - target)**2).sum()
# 4) figure out which weights caused us to miss
loss.backward()
# 5) change those weights
opt.step()
# 6) print our progress
print(loss.get().float_precision())
The loss indeed decreased!
You might wonder how encrypting everything impacts the decreasing loss. Actually, because the theoretical computation is the same, the numbers are very close to non-encrypted training. You can verify this by running the same example without encryption and with a deterministic initialisation of the model like this one in the model __init__:
with torch.no_grad():
self.fc1.weight.set_(torch.tensor([[ 0.0738, -0.2109],[-0.1579, 0.3174]], requires_grad=True))
self.fc1.bias.set_(torch.tensor([0.,0.1], requires_grad=True))
self.fc2.weight.set_(torch.tensor([[-0.5368, 0.7050]], requires_grad=True))
self.fc2.bias.set_(torch.tensor([-0.0343], requires_grad=True))The slight difference you might observe is due to the rounding of values performed while transforming to fixed precision. The default precision_fractional is 3 and if you get it down to 2 the divergence with clear text training increases, while it reduces if you choose precision_fractional = 4.
Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the movement toward privacy preserving, decentralized ownership of AI and the AI supply chain (data), you can do so in the following ways!
The easiest way to help our community is just by starring the Repos! This helps raise awareness of the cool tools we're building.
The best way to keep up to date on the latest advancements is to join our community! You can do so by filling out the form at http://slack.openmined.org
The best way to contribute to our community is to become a code contributor! At any time you can go to PySyft Github Issues page and filter for "Projects". This will show you all the top level Tickets giving an overview of what projects you can join! If you don't want to join a project, but you would like to do a bit of coding, you can also look for more "one off" mini-projects by searching for github issues marked "good first issue".
If you don't have time to contribute to our codebase, but would still like to lend support, you can also become a Backer on our Open Collective. All donations go toward our web hosting and other community expenses such as hackathons and meetups!
In [ ]: